Because your presence prevents the air behind you from flowing towards the fire, you’ve created an imbalance in the airflow.
Instead of having equal airflow from all sides of your fire, you now only have airflow from the opposite side.
This imbalance causes the smoke to move directly towards you. And this will happen regardless of where you move.
Furthermore, the more people you have around the fire, the more blockage of airflow you have. This makes the vacuum even stronger.
Another reason you seem to be a magnet for campfire smoke is because of tailwinds.
Even though you can’t see them, any movement whatsoever creates these one-directional vacuums. Whenever you create a tailwind, it essentially sucks the smoke right towards you.
And finally, Mother Nature itself plays a large role in the direction of the smoke. The wind itself carries the smoke in whatever direction it may be traveling.
Similar to water, smoke travels the path of least resistance.